Our Data

• As before, we’ll work with data from the 2022 Current Population Survey.
• We’ll look at California residents age 25-35 with at least an 8th-grade education.
  • This sample includes 2271 people.
  • We’ll think of them as having been sampled with replacement from a population.
  • That is, from the population of all California residents meeting these criteria.
  • That’s an approximation—the CPS is a complex survey—but it’s not too bad.

Unbiasedness

• These estimates are unbiased.
  • That is, their expected value is the population summary they’re meant to estimate.
  • So, loosely speaking, their sampling distribution is centered at the right location.
• We’ve seen this before in the binary-covariate case.
  • Subsample means are unbiased for subpopulation means—even conditional on \(X_1 \ldots X_n\).
  • This extends, via linearity of expectations, to combinations like the differences of subsample means.
• All this still generalizes, via linearity, to the non-binary covariate case.
  • It’s true for combinations with non-random coeffificents, like the average over years.
  • It’s true for combinations with random, but unbiased coefficients, like the average over people.

\[ \small{ \begin{aligned} &\mathop{\mathrm{E}}\qty[\hat \mu(x)] = \mathop{\mathrm{E}}\qty[\frac{1}{N_x}\ \sum_{i:X_i=x} Y_i] \qfor N_x = \sum_{i:X_i=x} 1 && \quad \textbf{is equal to} \quad \mu(x) = \frac{1}{m_x}\ \sum_{j:x_j=x} y_j \qfor m_x = \sum_{j:x_j=x} 1 \\ &\mathop{\mathrm{E}}\qty[\frac{1}{4}\sum_{x=9}^{12} \qty{ \hat \mu(x) - \hat\mu(x-1) }] && \quad \textbf{is equal to} \quad \frac{1}{4}\sum_{x=9}^{12} \qty{ \mu(x) - \mu(x-1) } \\ &\mathop{\mathrm{E}}\qty[\sum_{x=9}^{12} P_x \qty{ \hat \mu(x) - \hat\mu(x-1) }] \qfor P_x = \frac{N_x}{\sum_{x=9}^{12} N_x} && \quad \textbf{is equal to} \quad \sum_{x=9}^{12} p_x \qty{ \mu(x) - \mu(x-1) } \qfor p_x = \frac{m_x}{\sum_{x=9}^{12} m_x} \\ \end{aligned} } \]

The Average Over Years

1. To show unbiasedness, we expand our estimator into a linear combination of subsample means. \[ \frac{1}{4}\sum_{x=9}^{12} \qty{ \hat\mu(x) - \hat\mu(x-1) } = \frac{\hat\mu(12) - \hat\mu(8)}{4} \\ = \sum_x \alpha(x) \hat \mu(x) \qfor \alpha(x) = \begin{cases} \hphantom{-}\frac14 & \text{ if } x = 12 \\ -\frac14 & \text{ if } x = 8 \\ 0 & \text{ otherwise} \end{cases} \]

2. Then we calculate its expectation in this general form.

\[ \begin{aligned} \mathop{\mathrm{E}}\qty[\sum_x \alpha(x) \hat \mu(x)] &= \sum_x \alpha(x) \mathop{\mathrm{E}}\qty[\hat\mu(x)] && \text{ via linearity } \\ &= \sum_x \alpha(x) \mu(x) && \text{ because $\hat\mu(x)$ is unbiased } \\ \end{aligned} \]

3. Then we reverse our expansion to get something more familiar-looking.

\[ \begin{aligned} \sum_x \alpha(x) \mu(x) &= \frac{\mu(12) - \mu(8)}{4} && \text{plugging in specific coefficients} \\ &= \frac{1}{4}\sum_{x=9}^{12} \qty{ \mu(x) - \mu(x-1) } && \text{'untelescoping'} \end{aligned} \]

• In the middle, we have the general case.
  • A linear combination of subsample means with non-random coefficients is an unbiased estimator …
  • of the same linear combination of subpopulation means.
• At the top and bottom, we translate between this general form and a more familiar-looking one.

The Average Over Years, Faster

• We could, in fact, have done the whole thing without translation.

\[ \begin{aligned} \mathop{\mathrm{E}}\qty[\frac{1}{4}\sum_{x=9}^{12} \qty{ \hat\mu(x) - \hat\mu(x-1) }] &= \frac{1}{4}\sum_{x=9}^{12} \qty{ \mathop{\mathrm{E}}[\hat\mu(x)] - \mathop{\mathrm{E}}[\hat\mu(x-1)] } && \text{via linearity }\\ &= \frac{1}{4}\sum_{x=9}^{12} \qty{ \mu(x) - \mu(x-1) } && \text{via unbiasedness of subsample means }\\ \end{aligned} \]

• But we’ll need the general form later, for variance calculation, so we took this opportunity to introduce it.

The Average Over People

1. Again, we expand our estimator into a linear combination of subsample means. \[ \begin{aligned} \frac{\sum_{i:X_i \in 9 \ldots 12} \qty{ \hat\mu(X_i) - \hat\mu(X_i-1) }}{\sum_{i:X_i \in 9 \ldots 12} 1} &= \sum_{x \in 9 \ldots 12} P_x \ \qty{ \hat\mu(x) - \hat\mu(x-1) } \quad \text{ for } \quad P_x = \frac{N_x}{\sum_{x \in 9 \ldots 12}N_x} \\ &= \sum_x \hat \alpha(x) \hat \mu(x) \qfor \hat\alpha(x) = \begin{cases} P_{12} & \text{ if } x = 12 \\ P_{x} - P_{x+1} & \text{ if } x \in \{9 \ldots 11\} \\ -P_9 & \text{ if } x = 8 \end{cases} \end{aligned} \]

2. We calculate is expectation in this general form. There’s a new wrinkle: our coefficients are random. \[ \begin{aligned} \mathop{\mathrm{E}}\qty[\sum_x \hat\alpha(x) \hat \mu(x)] &= \sum_x \mathop{\mathrm{E}}\qty[\hat\alpha(x) \hat\mu(x)] && \text{ linearity of expectations } \\ &= \sum_x \mathop{\mathrm{E}}\qty{ \mathop{\mathrm{E}}\qty[\hat\alpha(x) \hat \mu(x) \mid X_1 \ldots X_n] } && \text{ law of iterated expectations } \\ &= \sum_x \mathop{\mathrm{E}}\qty{ \hat \alpha(x) \mathop{\mathrm{E}}[\hat \mu(x) \mid X_1 \ldots X_n] } && \text{ linearity of cond. expectations }\\ &= \sum_x \mathop{\mathrm{E}}\qty{ \hat \alpha(x) \mu(x) } && \text{ conditional unbiasedness of sample means} \\ &= \sum_x \mathop{\mathrm{E}}\qty{ \hat \alpha(x) } \mu(x) && \text{ linearity of expectations }. \end{aligned} \]

3. If our coefficients are ‘unbiased’ …

• i.e. if their expectations are the coefficients \(\alpha(x)\) in the population summary
• then we can reverse the expansion to get the population summary in familiar form.

\[ \begin{aligned} \sum_x \mathop{\mathrm{E}}\qty{\hat\alpha(x)}\mu(x) &= \sum_x \alpha(x) \mu(x) && \qqtext{if} \mathop{\mathrm{E}}\qty{ \hat \alpha(x) } = \alpha(x) \\ &= \sum_{x=9}^{12} p_x \qty{ \mu(x) - \mu(x-1) } \qfor p_x = \frac{m_x}{\sum_{x=9}^{12} m_x} && \qfor \alpha(x) = \begin{cases} p_{12} & \text{ if } x = 12 \\ p_{x} - p_{x+1} & \text{ if } x \in \{9 \ldots 11\} \\ -p_9 & \text{ if } x = 8 \end{cases} \end{aligned} \]

Does This Work Out?

\[ \begin{aligned} &\text{Does} \quad \mathop{\mathrm{E}}\qty{ \hat \alpha(x) } = \alpha(x) \qfor \hat\alpha(x) = \begin{cases} P_{12} & \text{ if } x = 12 \\ P_{x} - P_{x+1} & \text{ if } x \in \{9 \ldots 11\} \\ -P_9 & \text{ if } x = 8 \end{cases} \qqtext{and} \alpha(x) = \begin{cases} p_{12} & \text{ if } x = 12 \\ p_{x} - p_{x+1} & \text{ if } x \in \{9 \ldots 11\} \\ -p_9 & \text{ if } x = 8 \end{cases} \end{aligned} \qqtext{?} \]

• Yes, it does. We can show that \(\mathop{\mathrm{E}}[P_x]=p_x\) for all \(x\) via a sort of ‘two-stage sampling argument’.
  • First flip a coin to decide if \(X \in \{9 \ldots 12\}\) or not.
  • Then roll a die to decide which value you take within the group.
  • \(P_x\) is, conditional on the coin flip, a sample proportion.
  • So, it’s unbiased for the population proportion \(p_x\).
• This last part isn’t something I’d expect you to be able to do on an exam.
• But it is important to get a satisfying conclusion in this example.

Summarizing Unbiasedness

• We’ve shown that our two specific estimators are unbiased.
• More generally, we’ve shown that a linear combination of subsample means is unbiased …
  • for the same linear combination of subpopulation means, if the coefficients are non-random.
  • for an analogous linear combination of subpopulation means, using the expected coefficients, if they are random.

\[ \begin{aligned} \mathop{\mathrm{E}}\qty[\sum_x \alpha(x) \hat \mu(x) ] &= \sum_x \alpha(x) \mu(x) \\ \mathop{\mathrm{E}}\qty[\sum_x \hat\alpha(x) \hat \mu(x) ] &= \sum_x \mathop{\mathrm{E}}[\hat\alpha(x)] \mu(x) \\ \end{aligned} \]

Interval Estimation

• It’s nice to know that our estimators are unbiased. That’s a start. But we want to know more.
• We want to know what an individual sample actually tells us about a population summary.
• To do this, we’ll use a calibrated interval estimate, i.e. a confidence interval.
• This looks exactly like it did in the binary-covariate case we discussed earlier in the semester.
  1. We estimate the sampling distribution of our estimator.
  2. We draw arms on our point estimate wide enough to cover 95% of this distribution—95/100 purple dots.
• And, as before, we can use the bootstrap sampling distribution as our sampling distribution estimate.
• But then, looking at our two summaries of the average value of a year of high school, we see something.
• Our estimate of the average over years is much more precise. Its arms are much shorter. Let’s understand why.

Understanding Sources of Error

• To understand why our intervals look like they do, we think about an interval based on normal approximation.
• The interval’s width is determined by the estimator’s standard error, i.e the square root of its variance.

\[ \text{population summary} \in \text{point estimate} \pm 1.96 \times \text{standard error} \]

• To make sense of this, what we need is a variance formula. Here it is.

\[ \mathop{\mathrm{\mathop{\mathrm{V}}}}\qty[\sum_x \hat\alpha(x) \hat\mu(x)] = \textcolor{blue}{\sum_x \sigma^2(x) \times \mathop{\mathrm{E}}\qty[ \frac{\hat\alpha(x)^2}{N_x} ]} + \textcolor{red}{\mathop{\mathrm{\mathop{\mathrm{V}}}}\qty[ \sum_x \mu(x) \hat \alpha(x) ]} \qfor \sigma^2(x) = \mathop{\mathrm{\mathop{\mathrm{V}}}}[Y_i \mid X_i=x] \]

• This breaks the variance of our estimator down into two parts.
  • The first reflects the randomness of our subsample means.
  • The second reflects the randomness of our coefficients \(\hat\alpha(x)\).
• Note that if our coefficients aren’t random, i.e. \(\hat\alpha(x)=\alpha(x)\), the second term is zero.
• And often, even if they are random, the first term is much bigger.
• As a result, we can usually get away with thinking about the first part only.
  • Note that the second part is positive—it’s the variance of something.
  • Ignoring it gives us a lower bound on the actual variance of our estimator.

The Formula

• This two-part structure arises from the law of total variance.
  • We won’t get deep into the calculations here.
  • That’s ‘appendix stuff’ as far as this class is concerned.

\[ \small{ \begin{aligned} \mathop{\mathrm{\mathop{\mathrm{V}}}}\qty[ \sum_x \alpha(x) \hat \mu(x) ] &= \mathop{\mathrm{E}}\qty[ \mathop{\mathrm{\mathop{\mathrm{V}}}}\qty{ \sum_x \hat\alpha(x) \hat \mu(x) \mid X_1 \ldots X_n }] \quad &&+ \quad \mathop{\mathrm{\mathop{\mathrm{V}}}}\qty[ \mathop{\mathrm{E}}\qty{\sum_x \hat\alpha(x) \hat \mu(x) \mid X_1 \ldots X_n} ] \\ &= \mathop{\mathrm{E}}\qty[ \mathop{\mathrm{\mathop{\mathrm{V}}}}\qty{ \sum_x \hat\alpha(x) \hat \mu(x) \mid X_1 \ldots X_n }] \quad &&+ \quad \mathop{\mathrm{\mathop{\mathrm{V}}}}\qty[ \sum_x \hat\alpha(x) \mathop{\mathrm{E}}\qty{ \hat \mu(x) \mid X_1 \ldots X_n} ] \\ &= \mathop{\mathrm{E}}\qty[ \mathop{\mathrm{\mathop{\mathrm{V}}}}\qty{ \sum_x \hat \alpha(x) \hat \mu(x) \mid X_1 \ldots X_n }] \quad &&+ \quad \mathop{\mathrm{\mathop{\mathrm{V}}}}\qty[ \sum_x \hat\alpha(x) \mu(x) ] \\ &= \textcolor{blue}{\sum_x \mathop{\mathrm{E}}\qty[ \hat\alpha(x)^2 \mathop{\mathrm{\mathop{\mathrm{V}}}}\qty{\hat\mu(x) \mid X_1 \ldots X_n}]} \quad &&+ \quad \textcolor{blue}{\sum_x \sum_{x'} \mu(x) \mu(x') \mathop{\mathrm{Cov}}\qty[\hat\alpha(x),\ \hat\alpha(x')]} \\ &=\textcolor{blue}{ \sum_x \sigma^2(x) \times \mathop{\mathrm{E}}\qty[ \frac{\hat\alpha(x)^2}{N_x} ]} \quad &&+ \quad \textcolor{blue}{\sum_x \sum_{x'} \mu(x) \mu(x') \mathop{\mathrm{Cov}}\qty[\hat\alpha(x),\ \hat\alpha(x')]} \end{aligned} } \]

Variance Estimation

\[ \mathop{\mathrm{\mathop{\mathrm{V}}}}\qty[\sum_x \alpha(x) \hat \mu(x)] \ge \textcolor{blue}{\sum_x \sigma^2(x) \times \mathop{\mathrm{E}}\qty[ \frac{\hat\alpha(x)^2}{N_x} ] \quad \text{ where } \quad \sigma^2(x) = \mathop{\mathrm{\mathop{\mathrm{V}}}}[Y_i \mid X_i=x]} \]

• The lower bound has two quantities we don’t know.

  • The subpopulation variances \(\sigma^2(x)\).
  • The expected value of the ratio of the coefficient squared and the sample size, \(\hat\alpha(x)^2 / N_x\).

• But we can estimate both easily.

• The subsample variances are good esimates of the subpopulation variances \[ \sigma^2(x) \approx \hat\sigma^2(x) := \frac{1}{N_x} \sum_{i:X_i=x} \qty{Y_i - \hat\mu(x)}^2 \]

• The squared coefficient / subsample size ratio is usually a good estimate of its expectation.

\[ \frac{\hat\alpha(x)^2}{N_x} \approx \mathop{\mathrm{E}}\qty[\frac{\hat\alpha(x)^2}{N_x}] \]

• Plugging them in gives us an estimate of our summary’s variance.

\[ \widehat{\text{standard error}}^2 = \sum_x \ \frac{\hat\sigma^2(x)\hat\alpha(x)^2}{N_x} \]

The Average Over Years

\[ \hat\sigma_{\hat\theta}^2 = \sum_x \frac{\alpha^2(x) \ \hat\sigma^2(x)}{N_x} \]

where

\[ \begin{array}{l|lll} x & 8 & 9 & 10 & 11 & 12 & 13 & \ldots & 20 \\ \hline \alpha^2(x) & 0.06 & 0.00 & 0.00 & 0.00 & 0.06 & 0.00 & \ldots & 0.00 \\ \hat\sigma^2(x) & 492M & 444M & 113M & 603M & 803M & 1B & \ldots & 14B \\ N_x & 14 & 26 & 16 & 75 & 605 & 357 & \ldots & 71 \\ \frac{\alpha^2(x) \ \hat\sigma^2(x)}{N_x} & 2.20M & 0.00 & 0.00 & 0.00 & 83.01K & 0.00 & \ldots & 0.00 \\ \end{array} \]

Comparing Our Two Summaries

\[ \mathop{\mathrm{\mathop{\mathrm{V}}}}\qty[\sum_x \hat\alpha(x) \hat\mu(x)] \ge \mathop{\mathrm{E}}\qty[ \mathop{\mathrm{\mathop{\mathrm{V}}}}\qty{ \sum_x \hat\alpha(x) \hat \mu(x) \mid X_1 \ldots X_n }] = \sum_x \sigma^2(x) \times \mathop{\mathrm{E}}\qty[ \frac{\hat\alpha(x)^2}{N_x} ] \]

• Let’s think about what this means for our two examples. \[ \small{ \begin{aligned} \hat\theta_{\text{years}} &= \frac14\sum_{x=9}^{12} \qty{ \hat\mu(x) - \hat\mu(x-1)} = \sum_x \alpha(x) \hat \mu(x) \qfor \alpha(x) = \begin{cases} \hphantom{-}\frac14 & \text{ if } x = 12 \\ -\frac14 & \text{ if } x= 8 \\ 0 & \text{ otherwise} \end{cases} \\ \hat\theta_{\text{people}} &= \sum_{x=9}^{12} P_x \qty{ \hat\mu(x) - \hat\mu(x-1)} = \sum_x \hat \alpha(x) \hat \mu(x) \qfor \hat\alpha(x) = \begin{cases} \hphantom{-}P_{12} & \text{ if } x = 12 \\ P_{x} - P_{x+1} & \text{ if } x \in \{9 \ldots 11\} \\ -P_9 & \text{ if } x = 8 \end{cases} \end{aligned} } \]

• We’ve seen that our estimate of the average-over-years summary of the value of a year of high school has large variance.

• The problem was that it paired a relatively large weight–\(\alpha(x)=-1/4\)—with a very small sample size \(N_8=14\).

• And we know the average-over-people summary has an even larger variance. Let’s compare.

A Comparison

\[ \begin{array}{l|c|ccccc|c} & x & 8 & 9 & 10 & 11 & 12 & \sum \\ & N_x & 14 & 26 & 16 & 75 & 605 & 736 \\ & \hat\sigma^2(x) & 492M & 444M & 113M & 603M & 803M & \\ &\hline \\ \text{for}\ \ \hat\theta_{\text{people}} &\hat\alpha^2(x) & 0.00 & 0.00 & 0.01 & 0.52 & 0.68 & \\ &\frac{\hat\sigma^2(x)\alpha^2(x)}{N_x} & 44K & 3K & 45K & \textcolor{blue}{4.2M} & 897K & 5.2M \\ \hline \\ \text{for}\ \ \hat\theta_{\text{years}} &\alpha^2(x) & 0.06 & 0.00 & 0.00 & 0.00 & 0.06 & \\ &\frac{\hat\sigma^2(x) \alpha^2(x)}{N_x} & \textcolor{blue}{2M} & 0 & 0 & 0 & 83K & 2M \\ \\ \end{array} \]

• For both summaries, most of the variance comes from one term—from the variance of one subsample mean.
• For the average over years, it’s the \(x=8\) subsample, with a sample size of \(N_x=14\) and a coefficient of \(-1/4\).
• For the average over people, it’s the \(x=11\) subsample, with …
  • a sample size \(N_x=75\) that’s roughly 4x bigger.
  • but a coefficient of \(-0.72\) that’s roughly 3x bigger, too.
• Variance grows with the square of the coefficient divided by the sample size …
• So this suggests variance would be roughly \(3^2 / 4 \approx 2.25x\) bigger.
• That’s not far off what we get when we compare bootstrap variance estimates. That’s 2.3x bigger.

What About The Other Part?

\[ \mathop{\mathrm{\mathop{\mathrm{V}}}}\qty[ \sum_x \alpha(x) \hat \mu(x) ] = \mathop{\mathrm{E}}\qty[ \mathop{\mathrm{\mathop{\mathrm{V}}}}\qty{ \sum_x \hat\alpha(x) \hat \mu(x) \mid X_1 \ldots X_n }] \quad + \quad \mathop{\mathrm{\mathop{\mathrm{V}}}}\qty[ \mathop{\mathrm{E}}\qty{\sum_x \hat\alpha(x) \hat \mu(x) \mid X_1 \ldots X_n} ] \]

• What we’ve been thinking about so far is the first part of the variance.
  • That’s zero when the coefficients are constants, but not when they’re random.
  • Because this part was larger for the summary with random coefficients, that implied its variance would be larger.
  • But are we missing something? Are we underestimating how much larger it’ll be by ignoring the second part?
• Basically no.
  • We can estimate the variance of our summary—the whole thing—by the variance of bootstrap sampling distribution. It’s 5.1M.
  • Our estimate for the first part only — based on our table — was 5.2M.
  • That suggests that the second part should be roughly \(5.1M - 5.2M \approx -100K\).
  • That’s impossible—the part is the variance of something, so it can’t be negative—but it suggests it’s not very big.

Summary

• If what we’re estimating is a linear combination of the subpopulation means, with non-random coefficients \[ \small{ \theta = \sum_x \alpha(x) \mu(x) } \]

• Then the same linear combination of subsample means is unbiased. \[ \small{ \hat \theta = \sum_x \alpha(x) \hat\mu(x) \quad \text{ satisfies } \mathop{\mathrm{E}}[\hat\theta] = \theta } \]

• Its variance is determined by the variance in the subpopulation, the coefficients, and the subsample sizes. And we can estimate it. \[ \small{ \mathop{\mathrm{\mathop{\mathrm{V}}}}[ \hat\theta ] = \sum_x \sigma^2(x) \alpha^2(x) \times \mathop{\mathrm{E}}\qty[ \frac{1}{N_x} ] \approx \sum_x \hat\sigma^2(x)\alpha^2(x) \times \frac{1}{N_x} =: \hat\sigma_{\hat\theta}^2 } \]

• We can use this to construct a confidence interval based on normal approximation. \[ \small{ \theta \in \qty[ \hat \theta - 1.96\hat\sigma_{\hat\theta}, \ \hat \theta + 1.96\hat\sigma_{\hat\theta}] \quad \text{ with probability } \approx .95 \qfor \hat\sigma_{\hat\theta}^2 = \sum_x \hat\sigma^2(x)\alpha^2(x) \times \frac{1}{N_x} } \]

• Fundamentally, we’re going to get an imprecise estimate if a small subsample has a large weight.

• By imprecise, we mean a large standard error and a wide confidence interval.

• If we have random coefficients, the same is true: the formula estimates a lower bound on our estimator’s variance.